Communication Theory II 

Lecture 6: Review on Probability Theory (cont'd) 

Ahmed Elnakib, PhD 
Assistant Professor, Mansoura University, Egypt 



General Notes 



o Sections 

■ Attend in your section 

■ To not loose the section quiz, if you have a hard excuse, before the section, send 
me a message using the course website "moodal" describing your excuse 

■ No transfer is allowed till April 1 st . 

o Oral Exam 

■ Divide yourselves in the section to groups (3-5). Next week is the last week to 
register a group, otherwise you will be assigned to a group using alphabet order. 
Give the group names/section to your lecturer assistant 

■ In the reference section, include at least one related journal paper with an 
impact factor larger than 1.5 and note to its impact factor in the presentation 



Example of Previous Lecture 



Radar, a remote sensing system, operates by transmitting a sequence of pulses and has its receiver listen to 
echoes produced by a target (e.g., aircraft) that could be present in its surveillance area. 



Let the events A and B be defined as follows: 

A = { a target is present in the area under surveillance } 
A c = {there is no target in the area) 
B = { the radar receiver detects a target ] 

In the radar detection problem, there are three probabilities of particular interest: 

P[A] probability that a target is present in the area; this probability is called the 
prior probability. 

P[B\A] probability that the radar receiver detects a target, given that a target is 
actually present in the area; this second probability is called the probability 
of detection. 

P[i?|A c ] probability that the radar receiver detects a target in the area, given that there 
is no target in the surveillance area; this third probability is called the 

probability of false alarm. 



Given: 



P[A] = 


0.02 


P[B\A] 


= 0.99 


P[B\A Q ] 


= 0.01 



Required: 



P[A\B] 



The probability that a 
target is present in the 
surveillance area given that 
the radar receiver has made 
a target detection. 



Question 



o Does the relation P(6|/A)+P(e c |yA)=l? 

■ Since conditional probability satisfies the axioms of probability. Then, P(6|/\)+P(6 c |/\)=l 

■ Proof: A=QnA=(B\JB c )nA=(Bn A) \J(BcnA) OR P(Q\A) =P (O r\A)/P(A)=l 

P(A)=P(B \A)P(A)+P(B C | A)P(A) P((B C U B) nA)/P(A)=l 

1= P(B\A)+P(B C \A) P(B\A)P(A)+P(B C \ A)P(A)=P(A) 

o Does the relation P(6|/4)+P(e|4 c )=l? 

■ A contradicting case: A and 6 are independent then A c and B are independent. Why? 
P(6|>4)=P(e|>4 c )=P(B), if P(B)*0.5 then the relation does not hold 

■ Note that P(S|>4 c )+P(e c |>4 c )=l 

■ The only condition P(B\A)+P(B\A C )=1 is when P(B C \A)=P(B\A C ) and P(B\A)= P(B C \A C ) 



Lecture Outli 



Review on probability theory 

■ Some important distributions 

■ Characteristic function 

■ The Gaussian distribution 

■ The central limit theorem 



Discrete R.Vs Distributions 



1. Binomial: 

• Parameters: n (positive integer), p (0 < p < 1) 

• P.mX: p(k) = {l)p k {l - p) n ~ k (k = 0, 1, 2, . . . , n) 

• Expectation and variance: fi = np, a 2 = np(l — p) 

• Arises as: Distribution of number of successes in success/failure trials ("Bernoulli trials") 

5 

e.g. P(2H appears in a series of 5 tosses)=( ) p(H) 2 p(T) 3 
2. Bernoulli 

\-p if jc = 0 
Pxi x ) = \ p if x = 1 

0 otherwise 

E[X] = p 
var[X] = p( 1 -/>) 



Discrete R.Vs Distributions (cont'd) 



3. Poisson: 

• Parameter: A > 0 

• P.m.f.: p(k) = e~ x ^ (k = 0, 1, 2, . . .) 

• Expectation and variance: fi = X, a 2 = X 

• Exponential series formula: 5Z^=o nT = e * 

• Arises as: Distribution of number of occurrences of rare events, e.g., accidents, disasters. 

• Poisson approximation to binomial distribution: A binomial distribution in which n is 
large and p correspondingly small (so that np is of moderate size) is approximately Poisson with 
parameter A = np; i.e., if X has binomial distribution with parameters n and p as above, then 
P(X k) « e~ x \ k /h\, where A np. 
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Discrete R.Vs Distributions (cont'd) 

4. Geometric: 

• Parameter: p (0 < p < 1) 

• P.m.f.: p(k) = (1 - p) k ~ l p (k = 1, 2, . . .) 

• Expectation: /i = 

• Geometric series formula: Y2rv=o rU = T^r (\ r \ ^ -0 

• Arises as: Distribution of time (trial) of first success in a success/f allure trial sequence. 
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Example 1 



A binary source generates digits 1 and 0 randomly with probability 
P(l)=0.8and P(0)=0.2 

(a) Display graphically the sequential evolution of a four digit sequence, 
the results 

(b) What is the probability that all the four digits are ' 1 's 

(c) What is the probability that exactly a 'V will occur in a four digit 
sequence 

(d) What is the probability that at least two ' 1 ' will occur in a four digit 
sequence 



Solution 



0000 P(0000)=Po 4 

P(0001)=Po 3 (l-Po) 
P(0010)=Po 3 (l-Po) 

P(0011)=Po 2 (l-Po) 2 

P(0100)=Po 3 (l-Po) 

P(0101)=Po 2 (l-Po) 2 
P(0110)=Po 2 (l-Po) 2 

0111 P(0111)=Po(l-Po) 3 
1000 P(1000)=Po 3 (l-Po) 



0001 
0010 

0011 

0100 

0101 
0110 



1001 
1010 

1011 
1100 

1101 
1110 



P(1001 
P(1010 

P(1011 
P(1100 

P(1101 
P(1110 



■Po 2 (l-Po) 2 
--Po 2 (l-Po) 2 

■Po(l-Po) 3 
■Po 2 (l-Po) 2 

■Po(l-Po) 3 
■Po(l-Po) 3 



Binomial 
distribution 



P(a T will occur) 

=P(0001)+ P(0010)+P(0100)+ P(1000) 
=4{l-Po) Po 3 =4x0.8x0.2 3 =0.0256 

=(*) (1-Po) Po 3 



P(at least two Ts will occur) 

=P(two 'l's will occur) 
+ P(three 'l's will occur) 
+P(four 'l's will occur) 

=P(0011)+ P(0101)+ P(0110)+ P(1001)+P(1010)+ P(1100)+ 
+P(0111)+ P(1011)+ P(1101)+P(1110) 
+ P(1111) 

=( 4 2 ) (l-Po) 2 Po 2 + ( 4 3 ) (l-Po) 3 Po+(^) (1-Po) 4 

=6x0.8 2 x0.2 2 +4x0.8 3 x0.2 3 +lx0.8 4 =0.1536+0.0164+0.4096 
=0.579584 



1111 P(llll)=(i-Po) 4 = 0.8 4 =0.4096 
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Example 2 (section problem) 



A discrete memoryless channel is used to transmit binary data. The channel is discrete in that it is 
designed to handle discrete messages and it is memoryless in that at any instant of time the 
channel depends on the channel input only at that time. Owing to the unavoidable presence of 
noise in the channel, errors are made in the received binary data stream. The channel is symmetric 
in that the probability of receiving symbol 1 when symbol 0 is sent is the same as the probability of 
receiving symbol 0 when symbol 1 is sent. 

The transmitter sends 0s across the channel with probability p 0 and Is with probability p i . The 
receiver occasionally makes random decision errors with probability p; that is, when symbol 0 is sent 
across the channel, the receiver makes a decision in favor of symbol 1, and vice versa. 

Referring to Figure below, determine the following a posteriori probabilities: 

(a) The conditional probability of sending symbol Ao given that symbol Bo was received. 

(b) The conditional probability of sending symbol A x given that symbol Bl was received. 



ii 



Solution 

Assume we have a binary symmetric channel in that the probability of receiving symbol 1 when symbol 0 is sent is the same as the probability of 
receiving symbol 0 when symbol 1 is sent. For each bit we have: 

Prior probabilities: 

P(Ao)=probability that we send the symbol Ao='0' =Po 
P(A ^probability that we send the symbol A^' 1 ' =P 1 =l-Po 

Conditional Probabilities (probabilities of channel error): 

P(Z?olA ^probability of receiving the symbol Bo='0' given that we sent the symbol Al 
P(B 1 IAo)=probability of receiving Bl given that we sent the symbol Ao 



Note that from the axiom of probability (normalization probability) 
P(5olA 1 )+P(5 1 IA 1 )=l 
P(BolAo)+P(B 1 IAo)=l 

Assumptions (seen in the figure): 

Let ¥{Bo\A x )=p then P(5 1 IA 1 )=1-/? 

Since the channel is symmetric, V{Bo\A6)=p thus P( J B 1 IAo)=l-/? 
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l 0 





Solution (cont'd) 



Evidence: 

P(#o)=probability of receiving the symbol 'Bo' 
P(# ^probability of receiving the symbol 6 B X 9 

P(£o)=P(£olAo)P(Ao)+P(£olA !)P(A i)=( 1 -p)Po+p( 1 -Po) 



Ao 



A Posteriori Probabilities 

P(Aol2? 0 )=probability of sending symbol Ao given that symbol Bo was received 

Using Bayes' rule: P(Ao\B Q )=P(BolAo)P(Ao)/P(Bo) 
P(A 1 l5 1 )=probability of of sending symbol A 1 given that symbol Bl was received 



Channel error =error if a 6 0' is sent +error if a 'V is sent 

P(channel error)= PiAonB^+PiA^Bo) 

=P(£ 1 IA 0 )P(Ao) + PCtfolA^PCAj) 



Continuous R.Vs 



Uniform f , , 

E[X] = 
var[X] = 

Exponential fx( x ) ~ 

E[X] = 

varfX 2 ] 

3. Gaussian f , , 

var[X] 



b — a 



a < x < b 



\(a + b) 
1 1 



^ exp(-Ajt), * > 0 and /I > 0 

2 



1 x 2 2 

-exp [-(*-//) /2<j ], 



Jlna 



-QC < .Y < 3C 



= <T 



Rayleigh 



/xW = 


^exp( 






E[X] = 


aJn/2 


var[X] = 


H) 



x > 0 and <j> 0 



5. Laplacian 

/ x (jc) = — exp(— ), -co < x < qo and yl > 0 

E[X] = 0 
var[X] = 2//T 
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Characteristic function (moment generating 

function) 

o Formulate the probability law of the expectation of the nth moment using pdf: 
Hg(X)] = J j(x)f x (x) dx — g(X) = X» — E [X' J ] = J x%(*)dv 

—30 



Characteristic function (cont'd) 



o Another way is to use the characteristic function: 

■ The characteristic function of a continuous random variable X is defined as the 
expectation of the complex exponential function 

r 00 

0 x (v)=E(e~ ivx )= J_ f x (x)e lvx dx Where v is real and j=V z l 



We may also view the characteristic function of the random variable X as the 
Fourier transform of the associated probability density function f x (x), except for 
a sign change in the exponent 
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Characteristic function (cont'd) 



o /xMand o x (v) forming a Fourier-transform pair 

o We may obtain the moments of the random variable X from the by 
differentiate both sides of with respect to v a total of n times, and 
then set v = 0; we thus get the result 



^n 



dv 



n 



r 00 

-00 



Moment theorem 



n 



r n 



E[X'] = (-j)"~<Mv)l 



dv 



n 



V= 0 



17 



Example 



o Using the characteristic function, find the mean of the exponential distribution given below: 



A exp(-Ax), x > 0 
0, otherwise 



Solution 



4>(v) = I A exp(— Av) exp(jvjr) d.v 
J o 



E[X] = -jO' x (v) 



V = 0 



_ 1 
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The Gaussian (Normal) Distribution 



oThe Gaussian distribution stands out, by far, as the most commonly 
used distribution in the statistical analysis of communications systems, 

o Let X denote a continuous random variable; the variable X is said to be 
Gaussian if its pdf has the form: 



f x {x) = 



1 



J2n 



exp 



2a _ 



Where [x is the mean a is the standard deviation, a is positive 
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The Gaussian Distribution Properties 



o A Gaussian random variable is uniquely defined by specifying its mean and variance 
o Gaussianity is preserved by a linear transformation 

2 

Let X be a Gaussian random variable with mean ju and variance a . Define a new random 
variable 

Y = aX+b 

where a and b are scalars and a ^ 0. Then F is also Gaussian with mean 

E[7] = afA + b 

and variance 

2 2 

var[K] = a~ <j~ 
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The Gaussian Distribution Properties (cont'd) 



o The sum of independent Gaussian random variables is also a Gaussian random variable, 
whose mean and variance are respectively equal to the sum of the means and the sum of the 
variances of the constituent random variables 

Let X and Y be independent Gaussian random variables with means jux and //y, 
respectively, and variances a^ and ay , respectively. Define a new random variable 

Z = X+Y 

The random variable Z is also Gaussian with mean 

E[Z] = jU X + fiy 

and variance 

2 2 
var[Z] = a x + ay 
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Jointly Gaussian Random Variables 



Let X and Fbe a pair of jointly Gaussian random variables with zero means and variances 

2 2 

and ay, respectively. The joint probability density function of X and Y is completely 
determined by cr^, oy, and p, where p is the correlation coefficient p (x, y) = CQV [ xy ] 
Specifically, we have °* ay 

/^Ay) = cexp(-q(x 9 y)) 
where the normalization constant c is defined by 

1 

c = 



and the exponential term is defined by 

( 2 2\ 



Jointly Gaussian Random Variables (special cases) 



i 



oifp=0: /xj(^) = w7 v exp 



f 2 

x 



= fxWfyiy) 



y 



2 A 



2 2 

V 2 <J X 2 <JyJ 



If the random variables X and /are both Gaussian with zero mean and if they are 
also orthogonal (that is, E[XY] = 0), then they are statistically independent 
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The standard Gaussian Distribution 



O When // = 0 and cr 2 = 1, ► / x (jc) = -L- e xp^~ ) ► F x (x) = ~= J ex p(~0 dr 

Distribution is defined 
on its standard form 



o The Q-function 2 W = 1 - Fyf,*) 

o The Q-f unction, <2(x), is equal to the area covered by the tail of the probability 
density function of the standard Gaussian random variable X, extending from 
x to infinity Q( _ x) = x _ Q(x) 










0.6 






0.4 






/ 0.2 






1 1 




1 1 



-3-2-10123 

(b) 

3.10 The normalized Gaussian (a) distribution 
function and (b) probability density function. 
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The standard Gaussian distribution (Q-function) table 



0.0 


.5000 


.5040 


.5080 


.5120 


.5160 


.5199 


.5239 


.5279 


.5319 


.5359 


0.1 


.5398 


.5438 


.5478 


.5517 


.5557 


.5596 


.5636 


.5675 


.5714 


.5753 


U.Z 


.J /bO 


.joJZ 


.5871 


.jy 1U 




.5987 


.OUZO 


.0U04 


.olio 


£1 A 1 
.0141 


0.3 


.6179 


.6217 


.6255 


.6293 


.6331 


.6368 


.6406 


.6443 


.6460 


.6517 


0.4 


.6554 


.6591 


.6628 


.6664 


.6700 


.6736 


.6772 


.6808 


.6844 


.6879 


0.5 


.6915 


.6950 


.6985 


.7019 


.7054 


.7088 


.7123 


.7157 


.7190 


.7224 


0.6 


.7257 


.7291 


.7324 


.7357 


.7389 


.7422 


.7454 


.7485 


.7517 


.7549 


n i 
u. / 


. / JOU 


./Oil 


. /04Z 


lAll 
. /O / J 


11(\A 
. / /U4 


ll'XA 


11 &A 
. / /04 


11QA 

. 1 /y4 




. /OJZ 


0.8 


.7881 


.7910 


.7939 


.7967 


.7995 


.8023 


.8051 


.8078 


.8106 


.8133 


0.9 


.8159 


.8186 


.8212 


.8238 


.8264 


.8289 


.8315 


.8340 


.8365 


.8389 


1.0 


.8413 


.8438 


.8461 


.8485 


.8508 


.8531 


.8554 


.8577 


.8599 


.8621 


1.1 


.8643 


.8665 


.8686 


.8708 


.8729 


.8749 


.8770 


.8790 


.8810 


.8830 


1 .z 


88/1Q 


88AQ 


8888 
.oooo 




.o^Zj 


8Q/I/I 






8QCT7 
.oyy 1 


am ^ 


1.3 


.9032 


.9049 


.9066 


.9082 


.9099 


.91 15 


.9131 


.9149 


.9162 


.9177 


1.4 


.9192 


.9207 


.9222 


.9236 


.9251 


.9265 


.9279 


.9292 


.9306 


.9319 


1.5 


.9332 


.9345 


.9357 


.9370 


.9382 


.9394 


.9406 


.9418 


.9429 


.9441 


1.6 


.9452 


.9463 


.9474 


.9484 


.9495 


.9505 


.9515 


.9525 


.9535 


.9545 


1. / 


.bOj4 




.yj 15 


.bOoZ 


.yjy 1 




.yoUo 


.yoio 


.yoZj 


.yojj 


1.8 


.9641 


.9649 


.9656 


.9664 


.9671 


.9678 


.9686 


.9693 


.9699 


.9706 


1.9 


.9713 


.9719 


.9726 


.9732 


.9738 


.9744 


.9750 


.9756 


.9761 


.9767 


2.0 


.9772 


.9778 


.9783 


.9788 


.9793 


.9798 


.9803 


.9808 


.9812 


.9817 


2.1 


.9821 


.9826 


.9830 


.9834 


.9838 


.9842 


.9846 


.9850 


.9854 


.9857 


z.z 


Q8A1 

.yooi 


Q8A/1 

.yo04 


Q8A8 


.yo 1 1 


087^ 

.yo / j 


0878 
.yo /o 


Q88 1 


Q88/1 


Q887 


oeon 
.yoy\j 


2.3 


.9893 


.9896 


.9898 


.9901 


.9904 


.9906 


.9909 


.9911 


.9913 


.9916 


2.4 


.9918 


.9920 


.9922 


.9925 


.9927 


.9929 


.9931 


.9932 


.9934 


.9936 


2.5 


.9938 


.9940 


.9941 


.9943 


.9945 


.9946 


.9948 


.9949 


.9951 


.9952 


2.6 


.9953 


.9955 


.9956 


.9957 


.9959 


.9960 


.9961 


.9962 


.9963 


.9964 


2.7 


.9965 


.9966 


.9967 


.9968 


.9969 


.9970 


.9971 


.9972 


.9973 


.9974 


2.8 


.9974 


.9975 


.9976 


.9977 


.9977 


.9978 


.9979 


9979 


.9980 


.9981 


2.9 


.9981 


.9982 


.9982 


.9983 


.9984 


.9984 


.9985 


.9985 


.9986 


.9986 


3.0 


.9987 


.9987 


.9987 


.9988 


.9988 


.9989 


.9989 


.9989 


.9990 


.9990 


3.1 


.9990 


.9991 


.9991 


.9991 


.9992 


.9992 


.9992 


.9992 


.9993 


.9993 


3.2 


.9993 


.9993 


.9994 


.9994 


.9994 


.9994 


.9994 


.9995 


.9995 


.9995 


3.3 


.9995 


.9995 


.9995 


.9996 


.9996 


.9996 


.9996 


.9996 


.9996 


.9997 


3.4 


.9997 


.9997 


.9997 


.9997 


.9997 


.9997 


.9997 


.9997 


.9997 


.9998 



1 . The entries in this table, x say, occupy the range [0.0, 3.49]; the x is sample value of the random variable X. 

2. For each value of x, the table provides the corresponding value of the Q-function: 



Q{x) = J -Fjx) = -™=f exp(-f 2 /2) di 

J2k J r 



The Central Limit Theorem 



o Given certain conditions, the arithmetic mean of a sufficiently large number of 
iterates of independent random variables, each with a well-defined expected value 
and well-defined variance, will be approximately normally distributed, regardless of 
the underlying distribution 

o Provides the mathematical justification for using the Gaussian distribution as a 
model for an observed random variable that is known to be the result of a large 
number of random events 
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Let Y " = Z x ' where the X t are independent 
and uniformly distributed random variables 
on the interval from -1 to +1 
Suppose that we generate 20000 samples of 
the random variable Y n for n = 10, and 
then compute the probability density 
function of Y n by forming a histogram of 
the results 

The figure compares the computed 
histogram (scaled for unit area) with the 
probability density function of a Gaussian 
random variable with the same mean and 
variance 

The figure clearly illustrates that in this 
particular example the number of 
independent distributions n does not have to 
be large for the sum Y n to closely 
approximate a Gaussian distribution 
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Questions 
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